It is proposed that the observed near-surface inflows towards the active regions and sunspot zones provide a nonlinear feedback mechanism that limits the amplitude of a Babcock-Leighton-type solar dynamo and determines the variation of the cycle strength. This hypothesis is tested with surface flux transport simulations including converging latitudinal flows that depend on the surface distribution of magnetic flux. The inflows modulate the build-up of polar fields (represented by the axial dipole) by reducing the tilt angles of bipolar magnetic regions and by affecting the cross-equator transport of leading-polarity magnetic flux. With flux input derived from the observed record of sunspot groups, the simulations cover the period between 1874 and 1980 (corresponding to solar cycles 11 to 20). The inclusion of the inflows leads to a strong correlation of the simulated axial dipole strength during activity minimum with the observed amplitude of the subsequent cycle. This in agreement with empirical correlations and in line with what is expected from a Babcock-Leighton-type dynamo. The results provide evidence that the latitudinal inflows are a key ingredient in determining the amplitude of solar cycles.